{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPQdVyztj/FeUf/udtXiblg",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LC1332/Chat-Haruhi-Suzumiya/blob/main/notebook/%E8%A7%92%E8%89%B2%E6%95%B0%E6%8D%AE%E7%BB%9F%E8%AE%A1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Chat凉宫春日 Chat-Haruhi-Suzumiya\n",
        "\n",
        "[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)]()\n",
        "[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)]()\n",
        "[![Huggingface Gradio](https://img.shields.io/static/v1?label=Demo&message=Huggingface%20Gradio&color=orange)](https://huggingface.co/spaces/silk-road/ChatHaruhi)\n",
        "\n",
        "\n",
        "这个脚本主要用来统计每个角色的story数量\n",
        "\n",
        "---\n",
        "\n",
        "**Chat凉宫春日**是模仿凉宫春日等一系列动漫人物，使用近似语气、个性和剧情聊天的语言模型，\n",
        "\n",
        "<details>\n",
        "  <summary> 本项目由李鲁鲁，冷子昂，闫晨曦，封小洋，scixing，沈骏一，Aria Fei, 米唯实, 吴平宇, 贾曜恺等开发。 </summary>\n",
        "\n",
        "李鲁鲁发起了项目，并完成了最早的版本，在多个微信群实现了测试。完成了训练数据自动生成对话部分，设计了整体的路线，并撰写报告。\n",
        "\n",
        "冷子昂负责了每一个阶段的Gradio开发，以及每个部分的功能整合和架构设计。\n",
        "\n",
        "闫晨曦一开始将李鲁鲁的notebook重构为app.py，参与了WebUI的embedding部分重构整合。\n",
        "\n",
        "封小洋进行了中文转日文模型的选型，完成了针对台词抽取图片的工具。整合了声纹识别。即将继续参加台本工具的开发。\n",
        "\n",
        "scixing实践了VITS语音，完成了台词对应的语音抽取，实现了第一个版本的声纹分类。\n",
        "\n",
        "沈骏一实现了使用ChatGLM2 finetune实验\n",
        "\n",
        "Aria(Yaying Fei)对接了whisper到台本工具。即将继续参加台本工具的开发。\n",
        "\n",
        "米唯实实现了Chat哆啦A梦的分支版本\n",
        "\n",
        "吴平宇部署了ChatGLM2的训练程序，并提供了训练的计算资源。\n",
        "\n",
        "贾曜恺开发了一个Vue前端实现方案\n",
        "\n",
        "</details>"
      ],
      "metadata": {
        "id": "zUvfVW4wFXbJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GpCKeK8xFWW4",
        "outputId": "b8f65d9e-5379-4e1a-9d8b-3995b27dd831"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'Chat-Haruhi-Suzumiya'...\n",
            "remote: Enumerating objects: 11554, done.\u001b[K\n",
            "remote: Counting objects: 100% (1672/1672), done.\u001b[K\n",
            "remote: Compressing objects: 100% (1552/1552), done.\u001b[K\n",
            "remote: Total 11554 (delta 116), reused 1648 (delta 110), pack-reused 9882\u001b[K\n",
            "Receiving objects: 100% (11554/11554), 158.05 MiB | 15.01 MiB/s, done.\n",
            "Resolving deltas: 100% (2086/2086), done.\n",
            "Updating files: 100% (8099/8099), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/LC1332/Chat-Haruhi-Suzumiya"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q openai langchain"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ob-iD_KOFkL2",
        "outputId": "1532d433-152d-49d9-dd11-4068164bab98"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m73.6/73.6 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m21.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/Chat-Haruhi-Suzumiya/kyon_generator/\n",
        "!ls"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nfv5x6iQGBbC",
        "outputId": "39793bdc-eb82-4b09-c6fa-099bff95aa69"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/Chat-Haruhi-Suzumiya/kyon_generator\n",
            "chat2dialogue.py\t   data\t\t\t  synthesis_chat_method_foo.py\n",
            "ChatGPT_for_generation.py  dialogue2chat.py\t  synthesis_chat.py\n",
            "chatlog2dialogue.ipynb\t   dialogue2embedding.py  test_kyon_generator.ipynb\n",
            "chatlog2dialogue.py\t   story2chat.py\t  train.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "我们主要希望扫描所有文件夹之后\n",
        "\n",
        "对所有文件夹运行这样的指令\n",
        "\n",
        "```\n",
        "!python story2chat.py -story_folder \"/content/Chat-Haruhi-Suzumiya/characters/haruhi/texts\" \\\n",
        "                     -output ./output/chat_from_story.jsonl \\\n",
        "                     -role \"春日\" \\\n",
        "                     -other_names 凉宫 凉宫春日\n",
        "```\n",
        "\n",
        "然后统计生成的jsonl的行数，进行累加，\n",
        "\n",
        "让我们进行第一步"
      ],
      "metadata": {
        "id": "sWWR5O6mGJEV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 文件夹扫描\n",
        "\n",
        "请为我实现一段python程序\n",
        "\n",
        "我希望扫描/content/Chat-Haruhi-Suzumiya/characters/下有多少子文件夹\n",
        "\n",
        "然后打印成['子文件夹1','子文件夹2']的形式"
      ],
      "metadata": {
        "id": "PZnUt7vCGYpl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "folder = '/content/Chat-Haruhi-Suzumiya/characters/'\n",
        "subfolders = [f for f in os.listdir(folder) if os.path.isdir(os.path.join(folder, f))]\n",
        "\n",
        "print(subfolders)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ecUgIMaKGFvv",
        "outputId": "3416da1f-68a9-4d2b-8079-dc4797715ef8"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['tangshiye', 'weixiaobao', 'Megumi', 'murongfu', 'liyunlong', 'Luna', 'wangduoyu', 'Ron', 'jiumozhi', 'ayaka', 'personality-data', 'Snape', 'haruhi', 'Malfoy', 'xuzhu', 'xiaofeng', 'duanyu', 'Hermione', 'Dumbledore', 'zhongli', 'wanderer', 'wangyuyan', 'qiaofeng', 'hutao', 'raidenShogun', 'yaemiko', 'lilulu', 'yuqian', 'Harry', 'McGonagall']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "count = 0\n",
        "\n",
        "for folder in subfolders:\n",
        "    count += 1\n",
        "    if count % 5 == 0:\n",
        "        print( f\"'{folder}',\" ,end = '\\\\\\n')\n",
        "    else:\n",
        "        print( f\"'{folder}',\" , end = '')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UfD8D6hHHCbe",
        "outputId": "1d9fe127-f430-4bf1-dc48-83556120adcb"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'tangshiye','weixiaobao','Megumi','murongfu','liyunlong',\\\n",
            "'Luna','wangduoyu','Ron','jiumozhi','ayaka',\\\n",
            "'personality-data','Snape','haruhi','Malfoy','xuzhu',\\\n",
            "'xiaofeng','duanyu','Hermione','Dumbledore','zhongli',\\\n",
            "'wanderer','wangyuyan','qiaofeng','hutao','raidenShogun',\\\n",
            "'yaemiko','lilulu','yuqian','Harry','McGonagall',\\\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "folder_names = ['tangshiye','weixiaobao','murongfu','liyunlong',\\\n",
        "'Luna','wangduoyu','Ron','jiumozhi',\\\n",
        "'Snape','haruhi','Malfoy','xuzhu',\\\n",
        "'xiaofeng','duanyu','Hermione','Dumbledore',\\\n",
        "'wangyuyan','qiaofeng',\\\n",
        "'yuqian','Harry','McGonagall' ]\n",
        "\n",
        "role_names = ['汤师爷','韦小宝','慕容复','李云龙',\\\n",
        "'Luna','王多鱼','Ron','鸠摩智',\\\n",
        "'Snape','春日','Malfoy','虚竹',\\\n",
        "'萧峰','段誉','Hermione','Dumbledore',\\\n",
        "'王语嫣','乔峰',\\\n",
        "'于谦','Harry','Professor McGonagall' ]"
      ],
      "metadata": {
        "id": "r4Rsx5kEHXQL"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "已知\n",
        "\n",
        "```python\n",
        "folder_names = ['tangshiye','weixiaobao','murongfu','liyunlong',\\\n",
        "'Luna','wangduoyu','Ron','jiumozhi',\\\n",
        "'Snape','haruhi','Malfoy','xuzhu',\\\n",
        "'xiaofeng','duanyu','Hermione','Dumbledore','zhongli',\\\n",
        "'wanderer','wangyuyan','qiaofeng',\\\n",
        "'yuqian','Harry','McGonagall' ]\n",
        "\n",
        "role_names = ['汤师爷','韦小宝','慕容复','李云龙',\\\n",
        "'Luna','王多鱼','Ron','鸠摩智',\\\n",
        "'Snape','春日','Malfoy','虚竹',\\\n",
        "'萧峰','段誉','Hermione','Dumbledore','zhongli',\\\n",
        "'wanderer','王多鱼','乔峰',\\\n",
        "'于谦','Harry','Professor McGonagall' ]\n",
        "```\n",
        "\n",
        "\n",
        "我希望批量以\n",
        "\n",
        "```python\n",
        "python story2chat.py -story_folder \"/content/Chat-Haruhi-Suzumiya/characters/<folder_name>/texts\" \\\n",
        "                     -output ./output/<folder_name>_from_story.jsonl \\\n",
        "                     -role \"<role_name>\"\n",
        "```\n",
        "\n",
        "的形式调用脚本\n",
        "\n",
        "请用python为我实现。并且在运行之前，确认output文件夹的存在性，不存在则新建一个。"
      ],
      "metadata": {
        "id": "5iLU46HEIdYu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if not os.path.exists('./output'):\n",
        "    os.makedirs('./output')\n",
        "\n",
        "for folder, role in zip(folder_names, role_names):\n",
        "    cmd = f\"\"\"python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/{folder}/texts'\\\n",
        "     -output ./output/{folder}_from_story.jsonl \\\n",
        "     -role '{role}'\"\"\"\n",
        "    print(cmd)\n",
        "    os.system(cmd)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2b4U4x0LIXYa",
        "outputId": "b7fa1430-6e38-4faa-d963-4958bd51db55"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/tangshiye/texts'     -output ./output/tangshiye_from_story.jsonl      -role '汤师爷'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/weixiaobao/texts'     -output ./output/weixiaobao_from_story.jsonl      -role '韦小宝'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/murongfu/texts'     -output ./output/murongfu_from_story.jsonl      -role '慕容复'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/liyunlong/texts'     -output ./output/liyunlong_from_story.jsonl      -role '李云龙'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Luna/texts'     -output ./output/Luna_from_story.jsonl      -role 'Luna'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/wangduoyu/texts'     -output ./output/wangduoyu_from_story.jsonl      -role '王多鱼'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Ron/texts'     -output ./output/Ron_from_story.jsonl      -role 'Ron'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/jiumozhi/texts'     -output ./output/jiumozhi_from_story.jsonl      -role '鸠摩智'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Snape/texts'     -output ./output/Snape_from_story.jsonl      -role 'Snape'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/haruhi/texts'     -output ./output/haruhi_from_story.jsonl      -role '春日'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Malfoy/texts'     -output ./output/Malfoy_from_story.jsonl      -role 'Malfoy'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/xuzhu/texts'     -output ./output/xuzhu_from_story.jsonl      -role '虚竹'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/xiaofeng/texts'     -output ./output/xiaofeng_from_story.jsonl      -role '萧峰'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/duanyu/texts'     -output ./output/duanyu_from_story.jsonl      -role '段誉'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Hermione/texts'     -output ./output/Hermione_from_story.jsonl      -role 'Hermione'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Dumbledore/texts'     -output ./output/Dumbledore_from_story.jsonl      -role 'Dumbledore'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/wangyuyan/texts'     -output ./output/wangyuyan_from_story.jsonl      -role '王语嫣'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/qiaofeng/texts'     -output ./output/qiaofeng_from_story.jsonl      -role '乔峰'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/yuqian/texts'     -output ./output/yuqian_from_story.jsonl      -role '于谦'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/Harry/texts'     -output ./output/Harry_from_story.jsonl      -role 'Harry'\n",
            "python story2chat.py -story_folder '/content/Chat-Haruhi-Suzumiya/characters/McGonagall/texts'     -output ./output/McGonagall_from_story.jsonl      -role 'Professor McGonagall'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "folder_names = ['tangshiye','weixiaobao','murongfu','liyunlong',\\\n",
        "'Luna','wangduoyu','Ron','jiumozhi',\\\n",
        "'Snape','haruhi','Malfoy','xuzhu',\\\n",
        "'xiaofeng','duanyu','Hermione','Dumbledore','zhongli',\\\n",
        "'wanderer','wangyuyan','qiaofeng',\\\n",
        "'yuqian','Harry','McGonagall' ]\n",
        "\n",
        "我希望进一步统计所有的./output/{folder}_from_story.jsonl各自有多少行，以 \"角色名 行数\"的形式输出"
      ],
      "metadata": {
        "id": "A2z2RTjSJQX0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "count_lines = 0\n",
        "\n",
        "times = 0\n",
        "\n",
        "for folder, role in zip(folder_names, role_names):\n",
        "    file = f'./output/{folder}_from_story.jsonl'\n",
        "    with open(file) as f:\n",
        "        lines = len(f.readlines())\n",
        "        count_lines += lines\n",
        "    times += 1\n",
        "    if times % 3 == 0:\n",
        "        print(f'{role} {lines}')\n",
        "    else:\n",
        "        print(f'{role} {lines}', end = '\\t\\t')\n",
        "\n",
        "print('总共 ' + str(count_lines) + ' 行')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g_ONd8LxJGiG",
        "outputId": "d2a763de-92b2-4c3c-a744-bc0e0280baf7"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "汤师爷 75\t\t韦小宝 2394\t\t慕容复 158\n",
            "李云龙 931\t\tLuna 73\t\t王多鱼 115\n",
            "Ron 1423\t\t鸠摩智 86\t\tSnape 216\n",
            "春日 457\t\tMalfoy 163\t\t虚竹 197\n",
            "萧峰 246\t\t段誉 504\t\tHermione 1418\n",
            "Dumbledore 436\t\t王语嫣 202\t\t乔峰 190\n",
            "于谦 774\t\tHarry 2867\t\tProfessor McGonagall 169\n",
            "总共 13094 行\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "LlQN1LFoJmuC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "临时代码 ，重新抽取切分一下李云龙"
      ],
      "metadata": {
        "id": "GUg_usCGLnO5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "用utf-8格式读取/content/liyunlong_merge.txt\n",
        "\n",
        "通过\\n\\n split整个文档，\n",
        "\n",
        "以不同的文件名，txt格式，utf-8编码,保存所有的非空子字符串到/content/output中\n",
        "\n",
        "然后把/content/output中的所有txt文件打包压缩\n",
        "\n",
        "保存到/content/liyunlong.zip\n",
        "\n",
        "请用python为我实现"
      ],
      "metadata": {
        "id": "bUNlO1SSLqo6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import zipfile\n",
        "\n",
        "input_file = '/content/liyunlong_merge.txt'\n",
        "output_dir = '/content/output'\n",
        "archive_file = '/content/liyunlong.zip'\n",
        "\n",
        "if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "with open(input_file, 'r', encoding='utf-8') as f:\n",
        "    content = f.read()\n",
        "\n",
        "parts = content.split('\\n\\n')\n",
        "for i, part in enumerate(parts):\n",
        "    if part.strip() != '':\n",
        "        filename = os.path.join(output_dir, str(i) + '.txt')\n",
        "        with open(filename, 'w', encoding='utf-8') as f:\n",
        "            f.write(part)\n",
        "\n",
        "with zipfile.ZipFile(archive_file, 'w') as z:\n",
        "    for filename in os.listdir(output_dir):\n",
        "        file_path = os.path.join(output_dir, filename)\n",
        "        z.write(file_path, filename)\n",
        "\n",
        "print('Done!')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3ozqVv-mLqMv",
        "outputId": "7002eb9d-75b3-4f39-c260-1cfe90301bec"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Done!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "sYnWFe6-MFoj"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}